JOURNAL OF DISTANCE EDUCATION 
REVUE DE L’EDUCATION A DISTANCE 
2010 

VOL 24, No. 1, 107-122 


Management and Evaluation of a pan-Canadian Graduate 
Training Program in Health Informatics 


Marilynne Hebert and Francis Lau 


Abstract 


Eight Canadian universities partnered to establish a Collaborative Health 
Informatics PhD/Postdoc Strategic Training Program (CHPSTP). The 6-year goal 
was to increase research capacity in health informatics in Canada. Three cohorts 
of 20 trainees participated in the training, which included online Research 
Learning Experiences, annual face-to-face workshops and conferences. Different 
management models were explored to more closely align funding, user 
engagement and deliverables. Evaluation included measures of success at 
program, faculty and student levels. Data collection templates facilitated 
reporting and program management. Linkages between user engagement and 
funding should be built into the grant proposal. 


Resume 


Huit universites canadiennes se sont unies en partenariat pour mettre sur pied le 
Collaborative Health Informatics PhD/ Postdoc Strategic Training Program 
(CHPSTP). L'objectif sur six ans etait d'accroitre la capacite de recherche en 
informatique de la sante au Canada. Trois cohortes de 20 etudiants ont participe a 
la formation, laquelle comprenait des experiences de recherche en ligne, des 
ateliers annuels en face-a-face et des conferences. Differents modeles de gestion 
ont ete explores afin d'aligner le plus possible le financement, 1'engagement de 
l'utilisateur et les livrables. L'evaluation comprenait des mesures de la reussite au 
programme, au niveau des enseignants et des etudiants. Des grilles de collecte de 
donnees ont facilite le suivi et la gestion du programme. Des liens entre 
1'engagement de l'utilisateur et le financement devraient faire partie integrante de 
la demande de subvention. 


Background 

Health Informatics (HI), the use of computers and communications 
technology in health care delivery, administration and education has been 
steadily increasing over the past 30 years. Given the growing number of 
HI applications such as electronic health records and clinical decision 
support systems, the demand for expertise in planning for 
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implementation and evaluation of these systems has also expanded. HI 
graduate training has been identified as a high priority need in Canada, 
particularly since it has been difficult to achieve a critical mass for a 
program of study in HI (School of Health Information Science, 2001). 

In the fall of 2002, eight Canadian university teams (Victoria, Calgary, 
Waterloo, McMaster, To rcnto, McGill, Sherbrooke and Dalhousie) 
partnered to establish a Collaborative Health Informatics PhD /Postdoc 
Strategic Training Program (CHPSTP). This was made possible through a 
successful funding application to the Canadian Institutes of Health 
Research (CIHR) and Michael Smith Foundation for Medical Research 
(MSFMR) (Author, Author, 2002). The 6-year program was intended to 
increase research capacity in health informatics (HI) in Canada through: 

• jointly offering 4-5 intensive "Research Learning Experiences" 
(RLEs) to trainees not available in local programs; 

• strengthening collaboration among partner institutions in HI 
education and research; 

• building a virtual community engaged in HI research, education 
and practice; and 

• assuming a leadership /facilitation role as an academic HI 
community in Canada. 


Program Goals and Objectives 

The Program Goal was to create a shared resource of the knowledge and 
expertise in the eight HI research teams and through this to provide a 
wide range of HI research training opportunities to enhance the health 
system in ways not feasible within any single institution. Together the 
different perspectives and experiences of key mentors and stakeholders 
from across Canada could potentially take transdisciplinary HI research 
to a level well beyond what could be achieved locally. 

The curriculum and delivery methods were developed to overcome 
the barriers in access, diversity and depth faced by local programs. The 
use of e-learning and virtual presence technologies enhanced 
effectiveness of teaching, learning and collaboration, which previously 
had been done mostly in traditional face-to-face settings. Collaboration 
with stakeholder organizations such as health-care facilities and 
government agencies through the use of physical and virtual 
collaboratories were intended to ensure the relevance and accelerated 
pace of HI research. 
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The Program Objectives were to: 

a) increase the number of researchers at PhD and postdoctoral levels 
in Canada; 

b) establish a critical mass of HI researchers otherwise not achievable 
in local programs; 

c) build an effective collaborative research network to bridge the 
current and next generation of HI researchers; 

d) foster a culture of policy responsive researchers; 

e) explore new frontiers of HI research and 

f) address the growing demand for HI leadership in health within 
Canada. 

Program Delivery was planned in three iterations, with each cohort of 
trainees studying together for approximately 2 years. The CIHR funding 
rules for this grant stipulated that 70% of the $1.2 million was to be spent 
on trainee stipends. The funding was allocated annually to the Principal 
Investigator (PI) located at the University of Victoria, and then was 
distributed to the partner institutions. 

Of course what was planned and what unfolded changed over the 
three iterations. Program management, design and evaluation plus 
lessons learned are highlighted for each iteration of the program. These 
contributed to the overall lessons for management of large e-learning 
projects, in particular important issues around user engagement. 

Program Iteration 1 -Health Themes and Health Informatics Domains 
(January 2003-December 2004) 


Program Management 

A large gap existed between receiving program funding and accepting 
students into the program! Prior to accepting the first cohort of 20 
trOainees a program infrastructure was established. A part-time Program 
Director was hired to develop the processes and documentation required. 
The Program Director and one mentor from each university oversaw 
operational activities through a Program Management Committee (PMC). 
An Evaluator was hired part-time to assess how well the program met its 
objectives and CIHR reporting requirements. Responsibility for 
distributing the funds remained with the PI University of Victoria. Thirty 
percent of the annual funds were allocated to operational costs and the 
remaining 70% were divided equally among the 8 partners. 

As part of the accountability for receiving government funding as well 
as to provide overall strategic direction, an external Program Advisory 
Committee (PAC) was established in January 2003. PAC members were 
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identified among future employers of the graduates, students, HI 
industry, and academia. They met three times a year to provide external 
feedback on program design, implementation and evaluation. 

Program Design 

The first cohort of 20 trainees participated from January 2003-December 
2004. There were no shortages of trainees and partners were invited to 
enrdl additional trainees depending on local funding availability. 
PhD /Postdoc trainees had to be registered in one of the existing local 
graduate programs prior to participating in CHPSTP. In addition, while 
participating in these RLEs, trainees were also expected to enroll in 
required courses within their local programs. Training components 
included online Research Learning Experience (RLE) modules with 
virtual classes, annual on-campus workshops and face-to-face meetings at 
HI related conferences. 

HI competencies, identified through two nationally funded projects 
(School of Health Information Science, 2001; Covvey HD, Zitner D & 
Bernstein R, 2001) provided the basis for designing the RLEs. The RLEs 
also combined a unique delivery concept with shared collective expertise. 
Four RLEs were offered — Frontiers of HI Research, HI Virtual 
Community, HI Knowledge Transfer and HI Onsite Experience (Table 1). 

Table 1 . Summary of Research Learning Experiences 


RLE Title Purpose 


Delivery Method 


1. Frontiers 10 online sessions on-line discussion period 

of HI Research held every other week. preceded and followed a 

4-day onsite workshop 

Student expectations for 
participation were 
negotiated at each 
university. 

HI researchers and students 
brought together for 
discussions around leading 
edge HI research as well as 
strengthened the virtual 
community. 


Mentors lead discussions on 
their research. 
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RLE Title 

Purpose 

Delivery Method 

2. HI Research 

Mentors lead regular small 

Regular virtual and face-to-face 

Projects 

group meetings to help 
trainees produce tangible 

meetings. 


outputs. 

2 on-campus 3 day workshops 

3. Virtual HI 

Virtual community of 30 HI 

Cohort 1 : WebCT®, an 

Community 

researchers and graduate 

asynchronous course 


students across the country 

development tool 


meet monthly through 

• web-based synchronous 


asynchronous on-line 

presentations using Centra 


discussions and 

Symposium® Communication 


synchronous presentations. 

Cohort 2: a Web-based 
discussion forum (Moodle®) 


Monthly on-line trainee 

• virtual meeting groupware 


presentations 

(Elluminate®) 

• online critical review tool from 


Discussion of research topics, 
with a view to improving 
funding and publication 
success as well as 
collaboration opportunities. 

ShirWin 

4. On-Site Experience 

Students' offered exchange 

Face-to-face; short-term stay 


opportunities between 
universities. 

at exchange university 


Given the researchers' expertise, the program focus was narrowed to 
include six health domains — cancer research, healthy aging, 
public /population health, health policy and services, and cardiovascular 
and respiratory health. Selected crosscutting HI themes included 
knowledge management, intelligent health systems, telehealth and 
tele-learning, organizational informatics, e-health applications, and 
e-research applications. These themes were woven throughout the RLEs 
based on faculty and student interest. This demonstrates the breadth of 
topics that the trainees were exposed to as opposed to one or two research 
interests at the local university. 
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Program Evaluation 

The goals of the program and funder were partially fulfilled. A 
community of HI researchers was successfully developed with 
increasingly more formalized communication. A University of Calgary 
Master's student studied the development of this community of practice 
as part of her thesis work (Palacios, 2005). The funder's goals, to increase 
additional student funding and publications were also starting to be 
achieved. 

Program evaluations were conducted regularly through-out this 
iteration including post-course evaluation surveys for RLEs and 
face-to-face workshops. The possibility of offering trainees a certificate of 
completion was explored but rejected by the funding body. Each 
individual training program was free to provide certificates. The 
challenge was determining how much of the program had to be 
completed to receive a certificate — attend 80% of the web seminars and 
produce a manuscript? Or attend 75% and produce two manuscripts? 
This highlighted the challenge of determining what was considered 
adequate participation. Only 9/22 trainees submitted their progress 
reports for the overall CHPSTP Progress Report: April 2002-March 2004. 
However the list of deliverables for these 9 trainees was impressive with 
multiple conference presentations and publications. 

The CIHR also developed a "Performance Measurement and 
Evaluation Framework" that focused on increased opportunities for 
health research training and trainee deliverables that would not have 
been possible without this funding, e.g., publications and successful 
funding applications. 


Iteration 1 — Lessons Learned: 

• a broad range of faculty and trainee interests created challenges in 
sustainability 

• allocating funding on an equitable basis was not effective in 
guaranteeing participation or tangible outputs 

• cohesion could be enhanced by introducing smaller groups who 
were interested in common research areas 

• the Program Advisory Committee, while active, was not effective 
in setting the program's strategic direction. 
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Program Iteration 2: Cohort Model 
(January 2005-August 2006) 


Program Management 

Program Management was assumed by the PI as the Program Director 
with the support of a program coordinator from University of Victoria. A 
Program Management Committee (PMC) remained in place with one 
mentor from each institution to oversee all research training activities. 
The PMC held monthly virtual meetings to review progress. The 
Evaluator continued in her appointment to assess whether the program 
met the objectives and CIHR reporting requirements. 

A re-constituted Program Advisory Committee (PAC) with 6 members 
nominated by the PMC plus two non-voting members provided input to 
overall direction. The PAC met twice a year to review achievements and 
next steps. Each institution had to submit an annual performance report 
summarizing its activities for the year in order to receive further funding. 
UVic then submitted an aggregate report to CIHR each year. 

Funding Support 

Each partner institution was provided with $28k each year to support all 
CHPSTP related activities. This funding was intended to cover at least 
one trainee including his/her stipend and travel, as well as the 
administrative /travel /communication expenses for the mentors. Each 
partner site was required to submit a one-page plan to outline its 
projected expenditures for the year. Continued funding to each partner 
institution was contingent on satisfactory performance at the end of each fiscal 
year. In addition, each partner had to expend its entire stipend each year 
before receiving further allocation since CIHR discouraged carryover of 
funds. 

Each mini-cohort was also allocated $7,500 per year to support 
additional meetings, and the respective coordinators had to provide a 
1-page plan to outline the budget and activities for the 20 months ahead. 
The annual workshop was covered through central funding of $30K 
administered by University of Victoria. Another $7,500 was allocated to 
evaluation of the training program. 
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Participation and Level of Commitment 

In addition to a change in funding support which now required a plan for 
expected expenditures, the type of participation and level of commitment 
required in order to take part in this program were made more explicit for 
the second iteration. These are briefly outlined below: 

• The training program is made up of mentors, trainees, staff and 
collaborators. 

• There are three categories of trainees based on whether they are fully 
or partially funded through this program, or affiliates funded 
elsewhere. Different stipend amounts are provided depending on 
the type of trainee and whether he /she is full or part time in the 
program. (CIHR stipends are approximately $20k for PhD trainees 
and approximately $40k for fellows including benefits per year 
assuming full-time status. Practitioners are provided with up to a 
$25k stipend for buyout time over 20 months.) 

• Each trainee has to be nominated by a mentor from his/her home 
institution. The mentor who is responsible for the trainee has to 
participate in the program at the same time. All mentors and 
trainees have to enroll in one of the three mini-cohorts. 

• Additional meetings will be scheduled around HI related 
conferences, such as e-health, AMIA and Medlnfo conferences. All 
trainees and mentors will be encouraged to attend these events. 

To ensure the success of this program, every participant was expected 
to commit time and effort to take part in the activities and produce the 
outputs required. A summary of the commitments and expectations for 
all participants was developed and circulated, including the activities, 
deliverables and timelines for each role. 

Program Design 

The training program was intended to be closely aligned with the 
research areas of the mentors so their work could be linked to the trainees 
and across partner institutions. Given the breadth of research interests, 
three thematic areas based on the CIHR health domains and HI research 
themes of partners were suggested in order to focus this cohort: primary 
care informatics (PCI); e-health policy and evaluation (e-health); and 
knowledge management (KM). These three themes were only intended to 
provide a high-level categorization of relevant research areas to 
encourage focused collaboration among the mentors, yet were still 
considered sufficiently broad to allow diverse HI research and 
development to be undertaken by the trainees. 
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The basic structure of the training program consisted, again, of online 
Research Learning Experience (RLE) modules with regular virtual classes, 
annual on-campus workshops and face-to-face meetings at HI related 
conferences. Three mini-cohorts were established and coordinated by two 
mentors from two partner institutions. These coordinators were 
responsible for organizing and facilitating specific research training 
activities for each mini-cohort. They were to finalize the research training 
schedule in Fall 2004. With the help of their mentors, trainees were to 
submit a research training plan in January 2005 to outline their goals, 
activities, deliverables and timelines. A sample training plan was 
provided. 

University of Victoria was responsible for organizing the online 
sessions for the RLEs, while the mini-cohort coordinators led the sessions. 
In addition. University of Victoria coordinated and organized the annual 
on-campus workshop, which was hosted at one of the partner institutions 
in the spring. 

Evaluation: Tangible Outputs 

To meet the CIHR reporting requirements, tangible outputs were needed. 
Specifically, trainees were required to produce tangible outputs within 
their selected theme, which included a peer-reviewed publication and one 
other deliverable such as a research proposal, conference paper, peer- 
reviewed journal paper, research report, business plan or case study. Each 
trainee was to submit a one-page training plan when starting the 
program, then a one-page progress report at the end of year one , and a 
one-page summary report in year six after completion, to describe their 
accomplishments and how they had benefited from the program. 


Iteration 2 — Lessons Learned: 

• dividing into cohorts under broad umbrellas of interest was 
intended to assist in bringing focus to the research areas. This was 
a realistic approach given any one researcher would not be an 
expert in all HI areas. The trainees were still exposed to the range 
of HI research through the face-to-face meetings and on-line 
presentations. 

• local leadership for each of the mini-cohorts was inconsistent 

• rather than depending on the "good will" of participants who were 
busy in their own institutions, leadership required taking a more 
explicit stance as to what was required and aligning the funding 
with deliverables. 
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Program Iteration 3: Knowledge Translation (KT) 
for Best Practices in HI 


Program Management 

The final phase of this initiative was intended to engage mentors and 
current as well as former trainees and fellows in this program in 
knowledge translation (KT) to foster best practices in HI to improve the 
Canadian healthcare system. This KT cohort model has three parts - 
monthly real-time virtual seminars, moderated issues-driven 
asynchronous discussion forums, and supporting trainees /fellows to 
present papers at health IT conferences. These activities were intended to 
provide the foundation towards establishing a virtual CHPSTP 
community that links HI research with best practices. 

Program Design 

Eight seminars were delivered monthly from September 2006 to May 2007 
as a series of virtual classes via a real-time Web conferencing tool called 
Elluminate®. Each of the eight university partners was responsible for 
one of these sessions, which was presented by the mentors, trainees, 
alumni or guest speakers from that institution. These sessions were 
promoted widely to HI practice communities such as COACH, Canadian 
Society for Telehealth and other health professional associations. 

Four asynchronous discussion forums were held from September 2006 
to May 2007 using the online forum tool in MOODLE®. These online 
forums were moderated by designated mentors to discuss specific issues 
for a fixed time period of usually two to three weeks each. Everyone was 
encouraged to participate in these forums by posting comments, which 
were to be summarized by the group at the end as CHPSTP position 
papers. 

To enhance linkages between HI research and best practices, trainees 
and fellows were encouraged to submit papers to health IT conferences. 
The mentors were to work with trainees /fellows on formulating ideas, 
writing a paper and providing feedback for the paper submission. 
Trainees /fellows who had their papers accepted could qualify for 
financial support of up to $1,500 to attend the conference (until the 
funding pool was exhausted). Examples of such conferences included the 
International Symposium on Health Information Management Research 
Conference held by Dalhousie, the ImprovelT Workshop by University of 
Toronto, the Canadian Society for Telehealth Conference by University of 
Calgary, e-Health Conference by COACH-Infoway and the Spring AMIA 
conference. 
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Trainees /fellows were encouraged to work closely with their mentors 
to establish a training plan with specific objectives, tasks, timeline and 
deliverables. Similarly, alumni interested in working with mentors 
and/or peers were encouraged to put forth specific research plans and 
proposals in order to be included as part of the CHPSTP research training 
activities. 

Evaluation: Tangible Outputs 

Through specific KT activities this cohort was intended to produce 
tangible deliverables in the form of established best practices in HI that 
can have measurable impacts and improvements in the healthcare system 
in Canada. 


Iteration 3 — Lessons Learned: 

• it was easier for trainees and mentors to participate with specific 
deliverables being targeted 

• funding was tied to production, i.e., papers accepted for conference 
presentation, yet did not lead to any more or less papers produced 
by trainees/ fellows. 


Discussion 

End-user engagement (both trainees /fellows and mentors) was a critical 
factor in managing this e-learning project. During the first program 
iteration, participation was expected to occur as a result of the collegial 
nature of a multi-institutional grant. Mentors at each site were not paid 
but were expected to participate based on the fact they were listed as co- 
investigators on the funding application. Trainees were funded to 
participate and for the most part did so. For the few who did not fully 
participate in the RLEs or attend face-to-face meetings, other than gentle 
persuasion and reminders, there was no leverage to change the funding 
allocation or encourage increased participation. 

By the second program iteration (year three) the amount of time and 
effort required to develop, implement and sustain the program became 
more evident. Creating data collection templates facilitated reporting and 
managing the program. Data were collected to complete project reports to 
funding agencies as well as to plan /manage continued program 
development. 

A number of tensions emerged including one between developing an 
academic research program which faculty were comfortable with versus 
developing and supporting a research training program. A second tension 
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existed for trainees who were mandated to finish their required course 
credits at their home institutions and yet wanted to participate in 
additional research training. Both of these tensions interfered with 
promoting user engagement. 

Management was by exerting a positive influence to encourage 
participation. This was not successful among participants with competing 
interests and for whom the project did not resonate. 

The final iteration (years five and six) attempted to bring a closer 
connection between the academic and practice communities. The context 
had changed as more HI programs had developed across the country and 
needed faculty. The CHPSTP program had contributed to preparing PhDs 
for this increased number of jobs. 

Conclusion 

Evaluation activities included measures of success at program as well as 
individual faculty and student levels. Feedback from specific learning 
exercises such as face-to-face workshops, was useful for planning for 
future activities but less helpful for program management. However 
trainees /fellows and mentors strongly agreed that the annual face-to-face 
meeting should be held earlier in the year because this boosted user 
engagement when they made personal connections with participants. 

Overall, the lessons learned contribute to strategies for successful 
management and user engagement of future multi-university e-learning 
projects. Comparisons of expected evidence of training program success 
and actual outcomes were used to develop Management Strategies for 
consideration in the next e-learning program to ensure user engagement 
(Table 2). 

Table 2. Recommended Changes in Management Strategies. 


Expected Outcomes Actual Outcomes Changes in Mgmt. Strategies 


(a) Achieving stated 
goals regarding the 
number of PhD/Post- 
doc trainees enrolled 
in participating 
universities. 


The number of 
trainees/fellows remained 
steady, which indicates a 
sustained level of interest. 


a) Creating models for 
recognizing the contributions of 
the CHPSTP to graduate 
training are required so 
participation is not simply an 
“add on”. 
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Expected Outcomes Actual Outcomes Changes in Mgmt. Strategies 


(b) The successful 
transfer, uptake and 
impact of new 
knowledge and 
insights generated 
in selected health 
domains and HI 
research themes. 


(c) The formation of 
transdisciplinary HI 
research teams that 
span multiple 
institutions 


Program activities have 
successfully resulted in the 
creation of a virtual network 
that shares knowledge 
across sites. 

The program was unique 
and the only one in the 
country. 


Multi-disciplinary teams 
were not created as 
local research projects 
remained the focus at 
each partner site. 


a) Develop a formalized 
curriculum in order for the 
program to become more 
tangible and perceived as 
worthwhile, i.e., it cannot be 
seen as something 
participants do “off the side of 
their desks in their spare time”. 

b) each partner must be 
responsible for determining 
formal credits, either as courses 
or continuing education with 
professional associations like 
CMA/CNA, etc. 

c) the program partners may 
negotiate with COACH as part of 
certification of HI professionals 
or offer the program as a 
certificate program from one 
university 

d) each partner must be 
responsible for one major 
activity or one key tangible event 
like a workshop, a seminar 
series, creating the course 
credits, curriculum or liaising 
with industry, etc. (doing one 
seminar each round is not 
sufficient). 

a) each partner (including 
industry) must bring a 
problem that is large enough 
to create teams to work on it 
as a project 

b) the project should be 
something that can benefit the 
entire program not just their own 
students, e.g., creating a 
systematic review methods 
group/team for the network 
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Expected Outcomes Actual Outcomes Changes in Mgmt. Strategies 


(d) The existence of 
formalized 
collaborative 
relationships with 
stakeholder 
organizations and 
our effectiveness to 
translate HI research 
and practice 
knowledge as 
perceived by them 


A Program Advisory 
Committee (PAC) 
included members from 
stakeholder groups. 

Program meetings were 
held in conjunction with 
national HI and e-health 
conferences to encourage 
communication with 
stakeholder organizations. 


(e) Team member 
success, especially 
the trainees, in 
assuming key roles 
in the planning, 
design, 

implementation and 
evaluation of health 
information and IT 
infrastructures in the 
Canadian health 
system at the federal, 
provincial and 
regional levels. The 
program was intended 
to enrich the graduate 
experience and provide 
opportunities not 
available locally. 


The program lost focus 
on its goal when it became 
just another funding source 
for some partners/students 
who felt taking part in extra 
activities in return was a 
nuisance. 

Mentors quickly became 
busy doing their own things 
leaving the students to deal 
with any program activities 
and events. 


a) Create a closer tie to 
industry in order to ground the 
program in reality and remain 
relevant. 

b) Work with Infoway, CIHI and 
jurisdictions as partners to 
create lists of research 
priorities to work on. 

c) Create student teams to work 
these priorities as projects. 

d) have student teams 
co-present at the 
annual workshop. 

a) The partner agreement 
needs to be stronger, 

with both mentor and student 
signing the project form and 
and annually submitting a 
progress report, which is 
reviewed by the Program 
Committee (PMC) 

b) This strategy should 
translate into less time 
pursuing participants for their 
reports and may also result in 
suspending funding if reports 
are not submitted. 
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These strategies include: 

• making the partner agreement tighter; 

• making curriculum/ credit a requirement; 

• having closer ties to industry; 

• having each partner bring projects to the table; 

• make each partner responsible for one major activity each year as 
well as one aspect of the program. 

These strategies may not be acceptable to potential partners and trainees 
who have insisted on doing their own research and are too busy with 
coursework to do these projects. They should then find their own funding 
and not rely on this program. User engagement requires partners who are 
willing to work together on team projects side by side with the industry 
and the health professionals to solve real healthcare problems. 
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